Pattern-matching processing method and image processing apparatus

ABSTRACT

The present invention relates to a pattern-matching processing method based on left and right images of an object photographed in stereo. A left area and a right area are generated, each having a fixed pattern containing plural pixels extracted from the left and right images, respectively, and in the left area, a reference image is generated by calculating an interpolation pixel value to be used between two transversely adjacent pixels based on the average of the two pixel values, and by calculating an interpolation pixel value to be used between two vertically adjacent pixels based on the average of the plural pixel values surrounding the relating position. Similarly, in the right area, interpolation pixel values are calculated and a comparison image is generated. Then pattern matching is performed between the reference image and the comparison images generated sequentially by shifting the right original image pixel by pixel.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority of Japanese patent application number2002-223819, filed Jul. 31, 2002.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a pattern-matching processing methodand an image processing apparatus. More particularly, the presentinvention relates to a pattern-matching processing method and an imageprocessing apparatus that relates to a distance measuring systemadopting image processing, and in which pattern matching is performed bygenerating left and right matching images to which pixel interpolationhas been introduced from the photographed original images of an objecttaken by a stereo camera.

2. Description of the Related Art

Generally, the so-called stereoscopic distance measuring method iswidely known as a three-dimensional measuring technique adopting imageprocessing, in which correlation of a pair of left and right images ofan object photographed from different positions by a stereo cameracomposed of two cameras is found and the distance is found from theparallax of the same object using image processing based on theprinciple of the triangulation using the camera parameters such as theposition of the stereo camera and the focal length.

Such a distance measuring method is applied to, for example, distancemeasurement of an object located ahead of a vehicle such as a car. Astereo camera facing ahead is mounted in a vehicle and the distance toan object located ahead of the vehicle is measured based on thephotographed stereoscopic images of the object ahead of the vehicle.

The distance information of an object calculated based on left and rightimages photographed by a stereo camera is sent to, for example, avehicle control unit or a warning device, and is used to, for example,immediately stop the vehicle according to the measured distance or toinform the driver of the distance to the vehicle in the road ahead.

When such a stereoscopic distance measuring system is mounted in avehicle, the system is usually equipped with a pattern-matchingprocessing section, a distance measuring section, and an objectrecognition processing section for left and right images taken by astereo camera.

In the pattern-matching processing section, left and rightpattern-matching images in a small area are generated from the left andright original images taken by the stereo camera, and stereoscopicpattern matching is performed to find correlation between each otherbased on the left and right pattern-matching images.

Based on the matching results obtained by the stereoscopicpattern-matching processing, the difference between positions of pixelsin the left and right original images (that is, the parallax) relevantto the object photographed is measured, and the distance from the stereocamera to the object is calculated using the measured parallax and theparameters relating to the camera.

Based on the generated distance data, whether the object photographed inthe original image is, for example, a vehicle in the road ahead isjudged. The result of the judgment is sent to a vehicle control unitthat performs the vehicle traveling control, and so on, in order tosupport driving or to avoid danger.

By the way, the pattern-matching images generated in thepattern-matching processing section are composed of, for example, ninepixels, that is, a 3×3 pattern consisting of three transverse pixels andthree vertical pixels, which are cut out respectively from the left andright original images photographed by the stereo camera.

When pattern matching is performed on these pattern-matching images, ifit is assumed that the left original image is, for example, thereference image and the right original image is a comparison image, theright pattern-matching images, which are cut out from the right originalimage while being shifted pixel by pixel, are compared with the leftpattern-matching image cut out from the left original image. At thistime, a correlation coefficient of each pattern-matching image iscalculated and the degree of matching is judged according to themagnitude of the correlation coefficient.

When it is judged that the left pattern-matching image and the rightpattern-matching image match each other, the position of the centralpixel in the left pattern-matching image of the left original image andthe position of the central pixel in the right pattern-matching image ofthe right original image are found. The parallax in the transversedirection is found from the difference between the left pixel positionand the right pixel position. In addition, the parallax in the verticaldirection is also found in the same technique. If the parallax of theobject viewed from the stereo camera can be found, it is possible tocalculate the distance between the stereo camera and the object byreference to the parameters of the camera.

According to a technique using a technology in which, as describedabove, the three-dimensional position of an object ahead of a vehicle ismeasured, a matching position at which left and right pattern-matchingimages match each other is found by performing stereoscopic matchingbetween the left and right pattern-matching images in small areasextracted from the left and right original images photographed instereo, and the distance between the positions of pixels of thecorresponding left and right pattern-matching images is output as aparallax.

As the parallax obtained by processing the stereoscopic images areexpressed in the unit of pixels in the image, if a distance iscalculated from the parallax expressed in the unit of pixels by usingthe triangulation, the resolution is degraded for longer distances to anobject.

Because of the degradation of resolution, an error is produced in thecalculated distance. This error is ascribed to the fact that theparallax detected from images photographed by a stereo camera isexpressed in the unit of pixels and is also a main cause of thedegradation of resolution in distance measurement based on the principleof the triangulation. This error commonly occurs in a case where othermatching positions in images are found and may affect the judgment of anobject in the object recognition section significantly.

Conventionally, as a consequence, resolution is improved to secure theaccuracy in distance measurement by interpolating pixels in each leftoriginal image and right original image photographed in stereo,respectively, when pattern-matching processing is performed.

There are various known methods as a conventional pixel interpolationtechniques. In one of them, pixel interpolation is performed based oneight pixels located adjacently to the pixel at the position to beinterpolated. In this method, the pixel values of each of eight pixelsadjacently surrounding the position to be interpolated are averaged andthe resultant value is taken as a pixel value for the position to beinterpolated. Then, for the whole of the original image, pixelinterpolation is performed between two adjacent pixels successively.

This interpolation technique requires a tremendous amount of calculationbecause interpolated images are obtained after the interpolation pixelvalues to be interpolated are calculated for the whole of the originalimage. As a result, it takes a long time for the processing ofinterpolation; therefore, a more or less simplified interpolationtechnique is also adopted.

In the simplified pixel interpolation technique for obtaining aninterpolated image, two adjacent pixels are selected, as basicinterpolation processing, and an interpolation pixel is interpolatedsequentially between the two adjacent pixels, with the average of thepixel values relating to the two pixels being taken as a pixel value forinterpolation.

First, two transversely adjacent pixels in the original image areselected and a pixel is interpolated sequentially between the twoadjacent pixels, with the average of the pixel values of the two pixelsbeing taken as a pixel value for interpolation. This process, in which apixel is interpolated between the two transversely adjacent pixels inthe transverse row in the original image, is performed from the top ofthe original image to its bottom sequentially. After this, pixelinterpolation is performed between two vertically adjacent pixels in theoriginal image. By utilizing the interpolation pixels alreadyinterpolated between pixels in the transverse direction as well, pixelinterpolation with the seven adjacent pixels is performed between thetwo vertically adjacent pixels in the original image. According to thepixel interpolation technique described above, interpolated images areobtained.

In this simplified interpolation technique described above, which hasbeen adopted to obtain interpolation pixels, for two vertically adjacentpixels in the original image, the average value of the two adjacentpixels is taken as the interpolation pixel value and, as for thevertical direction, the average value of pixel values each of the sevenadjacent pixels has is taken as the interpolation pixel value, with thealready calculated interpolation pixels being also used, therefore, theamount of calculations required for the pixel interpolation technique isconsiderably reduced compared to the case where the pixel value of theposition to be interpolated is obtained from the average value of eightadjacent pixels.

However, a distance measuring apparatus to be mounted in a vehicle isrequired to deliver as accurately and speedily as possible the distancemeasurement data to be supplied for driving support and warning and,therefore, there remains a problem that the amount of calculations isstill large and much time is required, if the conventional or simplifiedpixel interpolation technique is used without changes made thereto forgenerating a pattern-matching image from a stereoscopic image.

The object of the present invention is, therefore, to provide apattern-matching processing method and an image processing apparatus, inwhich pattern matching is performed by introducing pixel interpolationto the left and right pattern-matching images in small areas extractedfrom the photographed original images of an object taken by a stereocamera.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned problem, the pattern-matchingprocessing method according to the present invention comprises an areagenerating step for generating left and right areas that define a fixedrange, respectively, from the left and right images photographed instereo, a pixel generating step for generating an interpolation pixelbetween pixels contained in the left or right area, and apattern-matching step for performing pattern matching on the left andright areas.

In the pattern-matching step, pattern matching is performed based on theleft and right areas containing the interpolation pixels, or patternmatching is performed based on the left area and the right area that hasbeen interpolated with the interpolation pixels.

Moreover, in the pixel generating step, pixel interpolation is performedbetween two vertically adjacent pixels relating to the position to beinterpolated, after pixel interpolation is performed on the left areaand the right area that contain at the center the pixel, the matchingposition of which has been specified by pattern matching based on theleft area and the right area, pixel interpolation is performed on theright area that contains at the center the pixel, the matching positionof which has been specified by pattern matching based on the left areaand the right area, or pixel interpolation is performed between twotransversely adjacent pixels in the left area or the right area.

In addition, in the pixel generating step, after pixel interpolation isperformed between two transversely adjacent pixels in the left area orthe right area, pixel interpolation is performed between two verticallyadjacent pixels at the position to be interpolated based on the averagevalue of plural pixels surrounding the position to be interpolated, andwhen the average value is calculated, the plural pixels surrounding theposition to be interpolated are weighted.

Then, in the pixel generating step, the average value is calculated bygiving weight of a value less than 1 to the already interpolated pixelsamong the plurality of pixels surrounding the position to beinterpolated, and pixel interpolation is performed starting from thepixel position at which the number of pixels surrounding the position tobe interpolated is largest, which is the target for which the averagevalue is calculated.

Moreover, in the pixel generating step, pixel interpolation is performedbased on two pixels adjacent to each other only in the transversedirection in the left area and the right area.

The image processing apparatus according to the present invention is astereoscopic distance measuring system that measures the distance to anobject that is photographed in images by performing pattern-matchingprocessing based on left and right images of the object photographed bya stereo camera, to which the above-described pattern-matchingprocessing method has been applied.

According to the image processing apparatus of the present invention,the distance to an object ahead of a vehicle can be accurately detected,and the period of time required for pattern-matching processing fordetection can be reduced and the distance information for drivingsupport and warning can be provided both accurately and speedily.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects and features of the present invention willbecome clear from the following description of the preferred embodimentsgiven with reference to the attached drawings, wherein:

FIG. 1A and FIG. 1B are diagrams that illustrate examples of patternmatching of images interpolated with pixels according to a firstembodiment of the present invention.

FIG. 2A and FIG. 2B are diagrams that illustrate examples of patternmatching of images interpolated with pixels according to a secondembodiment.

FIG. 3A and FIG. 3B are diagrams that illustrate examples of patternmatching of images interpolated with pixels according to a thirdembodiment.

FIG. 4A and FIG. 4B are diagrams that illustrate examples of patternmatching of images interpolated pixels according to a fourth embodiment.

FIG. 5 is a diagram that illustrates an example of generating apattern-matching image by using a first pixel interpolation techniqueaccording to the present embodiments.

FIG. 6 is a diagram that illustrates an example of generating apattern-matching image by using a second pixel interpolation techniqueaccording to the present embodiments.

FIG. 7 is a diagram that illustrates an example of generating apattern-matching image by using a third and a fourth pixel interpolationtechniques according to the present embodiments.

FIG. 8 is a diagram that illustrates an example of generating apattern-matching image by using a fifth pixel interpolation techniqueaccording to the present embodiments.

FIG. 9 is a diagram that illustrates a general block configuration of animage processing apparatus in a stereoscopic distance measuring systemusing left and right images.

FIG. 10A and FIG. 10B are diagrams that illustrate examples of left andright images used for pattern matching.

FIG. 11A and FIG. 11B are diagrams that illustrate how pattern matchingis performed using left and right images.

FIG. 12 is a diagram that illustrates pixel interpolation in aconventional image processing.

FIG. 13A and FIG. 13B are diagrams that illustrate a conventional pixelinterpolation based on adjacent pixels.

DETAILED DESCRIPTION OF THE INVENTION

In order to make the effects of the invention clearer, a general processof pattern-matching, relating to the present invention, will beexplained below.

An application example of a stereoscopic distance measuring methodconventionally used is shown in FIG. 9. The example shown schematicallyis a case where a stereo camera facing forward is mounted in a vehiclesuch as a car and the distance to an object ahead of the vehicle ismeasured based on photographed stereoscopic images.

A stereoscopic distance measuring system comprises a stereo cameracomposed of a set of two cameras, a left camera 1 and a right camera 2,mounted so as to face forward, and an image processing apparatus 3.

The image processing apparatus 3 comprises a pattern-matching unit 31, adistance measuring unit 32 and an object recognition logic unit 33. FIG.9 shows only the main parts for image processing, but there areprovided, although not shown here, an image input unit that performsinput processing on images of an object ahead of the vehiclephotographed by the left camera 1 and the right camera 2, an originalimage memory that stores the photographed images processed in the imageinput unit as original images, a distance image memory that storesdistance-related images generated by processing the original images, andso on.

The two cameras that constitute the stereo camera, that is, the leftcamera 1 and the right camera 2 are synchronized with each other and areCCD cameras, the shutter speed of which is variable, and one of the CCDcameras is used as a main camera that photographs a reference image forstereoscopic processing and the other is used as a sub camera thatphotographs a comparison image for stereoscopic processing, both havingthe same base line length and being arranged so that the their axesperpendicular to the image surface are parallel to each other.

The image input unit is provided with various functional circuits suchas an amplifier for processing two-system analog photographed imagesignals from the cameras 1 and 2, respectively, and an A/D converter orthe like for image processing, and moreover, it is further provided withan image adjusting circuit for electrically correcting slight deviationsin the optical positions of the cameras 1 and 2. Therefore, in the imageinput unit, images photographed by the cameras 1 and 2 are convertedinto digital image data with a fixed gradation of luminance and arestored in the original image memory after errors in the mechanicalmounting positions of each camera 1 and 2 are corrected by imageadjustment.

The pattern-matching unit 31 is composed of various circuits andmemories for processing such as an adder, a differentiator, an absolutevalue calculating circuit, and so on, and generates left and rightpattern-matching images in small areas from a pair of original images,that is, a left image and a right image stored in the original imagememory, and performs stereoscopic pattern-matching to find a mutualcorrelation according to each pattern-matching image.

The distance measuring unit 32 measures the distance between pixels(=parallax) in the original image, which is caused according to thedistance to the photographed object, based on the matching resultobtained in the pattern-matching unit 31, and generates the data ofdistance to the object photographed in the original image by referenceto camera parameters.

The object recognition logic unit 33 judges whether the objectphotographed in the original image is, for example, the vehicle aheadbased on the distance data generated in the distance measuring unit 32.The result of judgment is sent to a vehicle control device 4 thatperforms vehicle traveling control to support driving and avoid dangerand, in addition, it is also sent to a warning device 5 that warns thedriver of the distance to the vehicle ahead.

Examples of pattern-matching images are shown in FIG. 10, which aregenerated in the pattern-matching unit 31 in the image processingapparatus 3 configured as described above. In FIG. 10, apattern-matching image, which is cut out from an original imagephotographed by a stereo camera for pattern-matching processing, iscomposed of nine pixels in a 3×3 pattern having 3 columns and 3 rows.FIG. 10A shows the pattern of a pattern-matching image cut out from theleft original image photographed by the left camera 1, and FIG. 10Bshows the pattern of a pattern-matching image cut out from the rightoriginal image photographed by the right camera 2. In these figures,each pixel corresponds one element in a grid.

If, for example, the left original image is used as a reference imageand the right original image, as a comparison image, pattern-matching ofthese pattern-matching images is performed by comparing thepattern-matching image in FIG. 10A, which is cut out from the leftoriginal image, with the pattern-matching image in FIG. 10B, which iscut out from the right original image, each time being shifted pixel bypixel. Then, the correlation coefficient of each pattern-matching imageis calculated and the degree of matching is judged according to thecorrelation coefficient.

FIG. 11 shows how the degree of matching of each pattern-matching imageis judged. In FIG. 11 also, a left original image X_(L) is taken as areference image and a right original image X_(R), as a comparison image.These figures show a case where an object O ahead is photographed by astereo camera; FIG. 11A shows the left original image X_(L) photographedby the left camera 1 and FIG. 11B shows the right original image X_(R)photographed by the right camera 2. In the left original image X_(L), anobject O_(L) at the position viewed from the left camera 1 is shown andin the right original image X_(R), an object O_(R) at the positionviewed from the right camera 2 is shown, respectively. As the object Ois located a fixed distance ahead of the stereo camera, the positions ofthe object O in the images appear different.

Next, how to find the parallax between images by pattern-matching isdescribed in an example of a case where the right end line of the objectO is processed.

First, it is assumed that a pattern-matching image PT_(L) can beextracted from the left original image X_(L), which has a pixel on theright end line of the object O_(L) as its center, for example, p₂₂ ofthe 3×3 pattern in FIG. 10A. Then, a pattern-matching image PT_(R), thevertical position of which is the same as that of the pattern-matchingimage PT_(L), is extracted from the right original image X_(R). The nextpattern-matching image PT_(R) is extracted after the presentpattern-matching image PT_(R) having a 3×3 pattern is shifted each timepixel by pixel, for example, to the right in the transverse direction,while the vertical position is maintained unchanged.

Each time the pattern-matching image PT_(R) is extracted after beingshifted pixel by pixel, the correlation coefficient with respect to thepattern-matching image PT_(L) is found based on the nine pixel valuescontained in the pattern-matching image PT_(R). The position of theimage, which has the least correlation coefficient among those obtainedby shifting the pattern-matching image PT_(R) pixel by pixel, is judgedto match the pattern-matching image PT_(L) in pattern.

When the pattern-matching image X_(L) and the pattern-matching imageX_(R) are judged to match each other, a pixel position P_(L) of thecentral pixel p₂₂ in the pattern-matching image X_(L) in the leftoriginal image X_(L) and a pixel position P_(R) of a central pixel q₂₂in the pattern-matching image X_(R) in the right original image X_(R)are found. According to the principle of finding a horizontal parallax,it can be found from the difference (P_(L)−P_(R)) between the pixelposition P_(L) and the pixel position P_(R). A vertical parallax can befound in the same manner. If the parallax of the object O viewed fromthe stereo camera can be found, the distance between the stereo cameraand the object O can be calculated by reference to camera parameters.

According to the stereoscopic distance measuring method described above,the mutual matching position between the left and right pattern-matchingimages is specified by performing a stereoscopic matching on the leftand right pattern-matching images in the small areas extracted from theleft and right original images photographed in stereo, and the distancebetween pixels in the corresponding left and right pattern-matchingimages is output as a parallax.

However, the unit of the parallax obtained by processing stereoscopicimages is a pixel in the images. Therefore, if a distance is measured bytriangulation using a parallax expressed in units of pixels, resolutionis degraded for far objects.

The resolution is described by reference to FIG. 11A and FIG. 11B. Theposition of the right end line of the object O_(L) in the left originalimage X_(L) is found as the pixel position P_(L) and the position of theright end line of the object O_(R) in the right original image X_(R) isfound as the pixel position P_(R), and it is assumed that thosepositions are expressed by the number of pixels located to left thereofin the images as, for example, P_(L)=600 and P_(R)=500. Then, theparallax between the images is expressed as P_(L)−P_(R)=100.

Even though it is found that the distance calculated using thetriangulation method based on the above-mentioned parallax correspondsto 10 m and the pixel position P_(L) is 600, there is possibility thatthe position of the right end line of the object O_(L) in the leftoriginal image X_(L) is actually, for example, 600.5, and if this is thecase, the parallax should be 100.5. However, the distance calculatedbased on this will be, for example, 9.8 m, resulting in occurrence of anerror. Such an error is caused because the unit of the parallax obtainedfrom the images photographed by a stereo camera is a pixel and,therefore, according to the principle of the triangulation, theresolution in distance measurement is degraded and the judgment on theobject is considerably affected.

Conventionally, resolution is improved to maintain the accuracy indistance measurement by performing pixel interpolation on the leftoriginal image and the right original image photographed by a stereocamera, respectively, when pattern-matching processing is performed inthe pattern-matching unit 31. The conventional pixel interpolationtechnique is described below by reference to FIG. 12 and FIG. 13.

FIG. 12 shows a case where pixel interpolation is performed based oneight pixels adjacently surrounding the position to be interpolated. Inthis figure, the position to be interpolated is denoted by x₅ and eightpixels adjacently surrounding the position are denoted by x₁ to x₄, andx₆ to x₉ in the original image. When the position x₅ is interpolatedwith pixels, the pixel value of position x₅ is calculated based on thepixel values of the eight adjacent pixels and using the followingexpressionx ₅=(x ₁ +x ₂ +x ₃ +x ₄ +x ₆ +x ₇ +x ₈ +x ₉)/8Then, position x₅ is interpolated with the value. In some cases, eachpixel may be weighted. According to this interpolation technique, pixelinterpolation is performed sequentially between two adjacent pixels forthe whole of the original image.

According to this interpolation technique, it takes a tremendous amountof time to find the interpolation pixel values, and as a result,interpolation processing requires a long time, therefore, a simplifiedinterpolation technique is adopted, to deal with this problem, as shownin FIG. 13. FIG. 13A shows an example of an original image, but in thiscase it is assumed that the original image has 5×5 pixels forsimplification of description. Pixels x₁₁ to x₅₅ are arranged. FIG. 13Bshows an example of an interpolated image, in which an interpolationpixel has been interpolated between individual pixels that make up theoriginal image shown in FIG. 13A. The interpolated image now has a 9×9pixel array, which is the result of the interpolated pixels. In FIG.13B, the original pixel to be interpolated with pixels is shown in abold line frame and an interpolation pixel is shown in a thin lineframe.

The pixel interpolation technique to obtain the interpolated image shownin FIG. 13B is as follows. First, an interpolation pixel is interpolatedsequentially between two pixels arranged in the transverse direction inthe original image. Regarding interpolation pixel y₁₁ to be interpolatedbetween pixel x₁₁ and pixel x₁₂, the interpolation pixel value y₁₁ isobtained by the following expressiony ₁₁=(x ₁₁ +x ₁₂)/2Then, regarding interpolation pixel y₁₂ to be interpolated between pixelx₁₂ and pixel x₁₃, interpolation pixel value y₁₂ is obtained by thefollowing expressiony ₁₂=(x ₁₂ +x ₁₃)/2In this manner, pixel interpolation is performed sequentially betweentwo pixels adjacent to each other in the transverse direction in theoriginal image from the top end to the bottom end.

After pixel interpolation has been performed between two pixels adjacentto each other in the transverse direction in the original image, pixelinterpolation is performed between two pixels adjacent to each other inthe vertical direction in the original image. First, regardinginterpolation pixel z₁₁ to be interpolated between pixel x₁₁ and pixelx₂₁, interpolation pixel value z₁₁ is obtained by the followingexpression using already obtained transverse interpolation pixels y₁₁and y₂₁ adjacent to pixel position z₁₁z ₁₁=(x ₁₁ +y ₁₁ +x ₂₁ +y ₂₁)/4Then, regarding pixel position z₁₂ to be interpolated, interpolationpixel value z₁₂ is obtained by the following expression using alreadyobtained interpolation pixels y₁₁, y₂₁ and z₁₁z ₁₂=(x ₁₁ +y ₁₁ +x ₁₂ +z ₁₁ +x ₂₁ +y ₂₁ +x ₂₂)/7In this manner, pixel interpolation with seven adjacent pixels isperformed between two pixels adjacent to each other in the verticaldirection in the original image using the interpolation pixels alreadyinterpolated between pixels in the transverse direction. Theinterpolated image shown in FIG. 13B is obtained by performing pixelinterpolation on the original image shown in FIG. 13A using the pixelinterpolation technique described above.

According to the pixel interpolation technique shown in FIG. 12, thepixel value of the position to be interpolated is obtained from theaverage value of the eight adjacent pixels. Compared to this pixelinterpolation technique, according to the pixel interpolation techniqueadopted in order to obtain the interpolated image shown in FIG. 13B, theaverage value of two adjacent pixels is taken as an interpolation pixelvalue to be interpolated between the two pixels adjacent to each otherin the transverse direction in the original image, and the average valueof seven adjacent pixels including the already calculated interpolationpixels is taken as an interpolation pixel in the vertical direction,therefore, the amount of calculations required for the pixelinterpolation technique shown in FIG. 13B is small.

However, a distance measuring device to be mounted in a vehicle isrequired to be able to provide distance information for driving supportand warning with high precision in measurement and also able tocalculate as speedily as possible. When pattern-matching images aregenerated from stereoscopic images, if the conventional pixelinterpolation technique is used without any changes made to it, therestill remains a problem that the amount of calculations is large andmuch time is required.

According to the pattern-matching processing method and the imageprocessing apparatus relating to the present invention, the period oftime required for performing pattern-matching can be reduced byimproving the pixel interpolation technique for the left and rightimages for pattern-matching in small areas extracted from thephotographed original image of an object taken by a stereo camera, andby reducing the amount of time required for generating pattern-matchingimages.

Next, the embodiments of the pattern-matching processing system and theimage processing apparatus according to the present invention aredescribed below by reference to FIG. 1 to FIG. 8, using an example casewhere a pattern-matching processing is adopted, into which pixelinterpolation for the photographed original images of an object taken bya stereo camera is introduced, in a stereoscopic distance measuringsystem.

In a conventional stereoscopic distance measuring system, prior topattern-matching processing, left and right original images photographedby a stereo camera are turned into interpolated images with doubledresolution by performing pixel interpolation on the whole of theoriginal images, and pattern matching is performed based on the left andright interpolated images. This pattern-matching processing requires atremendous amount of calculations and time for generating interpolatedimages.

According to the pattern-matching processing in the first embodiment,which is applied to the stereoscopic distance measuring system relatingto the present invention, the amount of calculations and the period oftime required for generating interpolated images are reduced while theaccuracy in distance is improved by generating pattern-matching imagesthat have been interpolated with pixels and by performing patternmatching on the left and right pattern-matching images only when patternmatching is performed.

In the first embodiment, when pattern-matching processing is performedbased on left and right original images photographed by a stereo camera,pixel interpolation is performed on small areas cut out for patternmatching, for example, patterns of 3×3, that is, nine pixels, as shownin FIG. 10A and FIG. 10B.

FIG. 1 shows examples of pattern-matching images after pixelinterpolation has been performed on patterns of 3×3, that is, ninepixels, cut out from left and right original images. FIG. 1A shows astate in which pixel interpolation has been performed on nine pixels p₁₁to p₃₃ in FIG. 10A, and FIG. 1B shows a state in which pixelinterpolation has been performed on nine pixels q₁₁ to q₃₃,respectively. In the figures, the pixels that have been interpolatedwith pixels are shown in bold line frames.

The conventional pixel interpolation technique shown in FIG. 13B isadopted for generating the pattern-matching images shown in FIG. 1.First, the reference image for pattern-matching processing is generatedby performing pixel interpolation between two transversely adjacentpixels, when nine pixels p₁₁ to p₃₃ are specified as a small areapattern shown in FIG. 10A. As a result, six interpolation pixels a₁₁ toa₃₂ are obtained. Next, from the average value of seven pixels based onnine pixels p₁₁ to p₃₃ and the already obtained six interpolation pixelsa₁₁ to a₃₂, 10 interpolation pixels b₁₁ to b₂₅ in the vertical directionare obtained, and the left pattern-matching image shown in FIG. 1A isthus generated.

Moreover, regarding nine pixels q₁₁ to q₃₃ shown in FIG. 1B making upthe comparison image, six interpolation pixels c₁₁ to c₃₂ in thetransverse direction and 10 interpolation pixels d₁₁ to d₂₅ in thevertical direction are obtained in the same pixel interpolationtechnique, and the right pattern-matching image shown in FIG. 1B isgenerated.

After the left pattern-matching image and the right pattern-matchingimage are thus generated, pattern matching is performed on the 5×5 pixelpatterns based on these images. In pattern matching, for example, pixelsp₁₁, b₁₁, p₂₁, b₂₁, p₃₁, . . . , in the left pattern-matching image arecompared with pixels q₁₁, d₁₁, q₂₁, d₂₁, q₃₁, . . . , in the rightpattern-matching image, and each difference between corresponding pixelvalues is obtained, respectively. In this manner, each differencebetween corresponding pixel values is obtained for the whole of the 5×5pixel patterns, and a correlation coefficient between the left and rightpattern-matching images is found.

Next, three vertically adjacent pixels, which will be obtained byshifting the pattern by one pixel in the transverse direction, are cutout from the right original image and, based on these three verticallyadjacent pixels and original pixels q₁₃, q₂₃ and q₃₃, interpolationpixels c₁₃, d₁₆, c₂₃, d₂₆ and c₃₃, though not shown here, are added tothe right end of the right pattern-matching image shown in FIG. 1B. Inthis manner, a right pattern-matching image, which is composed of 5×5pixel pattern containing the newly interpolated pixels, that is, c₁₁ toc₃₃, and which is shifted by one pixel from the previous image, isgenerated as the next comparison image. Then, pattern matching isperformed between the left pattern-matching image shown in FIG. 1A,which is the reference image, and the right pattern-matching image newlygenerated, which is the next comparison image.

A new right pattern-matching image of 5×5 pixel pattern, which will bethe next comparison image, is generated by performing pixelinterpolation each time so that a 5×5 pixel pattern shifted by one pixelis generated sequentially in accordance with the 3×3 pixel pattern.Then, it is possible to judge that the right pattern-matching image,which has the least correlation coefficient in comparison between eachright pattern-matching image and the left pattern-matching image, isbest matched in the transverse direction and to specify the transverselymatching position in the original image. In addition, regarding thevertical direction, if the right pattern-matching image of 5×5 pixelpattern, which includes interpolation pixels obtained by shifting theright pattern-matching image, which is the comparison image, by onepixel in the vertical direction, is generated sequentially, thevertically matching position can be specified.

In the pattern matching processing according to the first embodiment, asdescribed above, the right pattern-matching image of 5×5 pixel patternis generated sequentially by pixel interpolation during processing,therefore, the amount of calculations and the period of time requiredfor pattern-matching processing can be reduced, with doubled resolutionbeing achieved, compared to the case where the original image is justinterpolated with pixels to achieve doubled resolution.

Although both the left and the right pattern-matching images shown inFIG. 1A and FIG. 1B are generated for the pattern-matching processing ofthe left and right 5×5 pixel patterns interpolated with pixels,according to the pattern-matching processing in the second embodiment, a3×3 pixel pattern cut out from the left original image is used, withoutany changes being made to it, as the left pattern-matching image, thatis, the reference image, and a 3×3 pixel pattern cut out from the rightoriginal image is interpolated with pixels so that a rightpattern-matching image of 5×5 pixel pattern, which will be a comparisonimage, is generated. FIG. 2A shows the left pattern-matching image,which is the 3×3 pixel pattern cut out from the left original imagewithout any changes made to it, and FIG. 2B shows the rightpattern-matching image, which is a 5×5 pixel pattern interpolated withpixels. The pixel interpolation technique for generating the rightpattern-matching image is the same as that shown in FIG. 1A and FIG. 1B.

In the pattern matching using the left and right pattern-matching imagesshown in FIG. 2A and FIG. 2B, 3×3, that is, nine pixels are used in thepattern-matching processing. Pixels p₁₁, p₂₁, p₃₁, . . . , p₃₂ in theleft pattern-matching image are compared with pixels q₁₁, q₂₁, q₃₁, . .. , q₃₃ in the right pattern-matching image. By this comparison,correlation coefficients between the left and right pattern-matchingimages are found.

Next, interpolation pixels c₁₃, d₁₆, c₂₃, d₂₆ and c₃₃ are added in thesame manner as that shown in FIG. 1A and FIG. 1B for generating a rightpatter-matching image, which will be the next comparison image. In thismanner, a right pattern-matching image, which is a 5×5 pixel patternshifted by one pixel and includes the newly added interpolation pixels,that is, a pixel pattern of pixels c₁₁ to c₃₃, is generated as the nextcomparison image. Then, nine pixels p₁₁, p₂₁, p₃₁, . . . , p₃₂ in theleft pattern-matching image shown in FIG. 2A, which serves as thereference image, are compared with nine pixels c₁₁, c₂₁, c₃₁, . . . ,c₁₃, c₂₃, c₃₃ in the right pattern-matching image, which has been newlygenerated and serves as the next comparison image. As a result,correlation coefficients can be found.

While a 3×3 pixel pattern of the left original image is used without anychanges made to it as the left pattern-matching image serving as thereference image, regarding the right pattern-matching image, whichserves as a comparison image, pixel interpolation is performed each timeto generate the right pattern-matching image, which is a 5×5 pixelpattern serving as the next comparison image, so that a 5×5 pixelpattern shifted by one pixel is generated sequentially. Then, by thecomparison between each right pattern-matching image and the leftpattern-matching image, the right pattern-matching pattern image withthe least correlation coefficient is judged to be matched in thetransverse direction and the transversely matching position in theoriginal image can be specified.

Regarding the vertical direction, the right pattern-matching imageserving as a comparison image is shifted by one pixel in the verticaldirection and the right pattern-matching image of 5×5 pixel pattern isgenerated sequentially as a result by using interpolation pixels d₁₁ tod₂₅. In this manner, the vertically matching position can be specified.

In the second embodiment, as described above, when pattern matching isperformed, pixel interpolation is performed only for generating theright pattern-matching image serving as a comparison image and the rightpattern-matching image of 5×5 pixel pattern is generated sequentially,therefore, it is possible to further reduce the amount of calculationsand the period of time required for pattern-matching processing whileachieving doubled resolution, because pixel interpolation is notperformed for generating the left pattern-matching image, compared tothe case where pixel interpolation is performed so that the originalimage achieves doubled resolution without any changes made to it.

In the pattern-matching processing according to the first and secondembodiments, which have been described above by reference to FIG. 1 andFIG. 2, all of the pattern matching between the left and right patternmatching images is performed using the images interpolated with pixels.However, according to the third embodiment, in order to improve theefficiency of pattern-matching processing, after or if a mutuallymatching position can be approximately specified using normal 3×3 pixelpatterns, a pattern-matching image is generated by performing pixelinterpolation on peripheral pixels around the specified position, forexample, a small area within a pixel distance of the specified position.

It is assumed that the pixel q₁₂ is specified as the matching positionfor the right pattern matching as a result when a normalpattern-matching processing is performed on the left and right 3×3 pixelpatterns shown in FIG. 10A and FIG. 10B, which have been cut out fromthe left and right original images. However, even though it is assumedthat the pixel q₁₂ is specified as the matching position, the resolutionin matching remains unchanged as before and all that can be said is thatthe position has been specified approximately. If the position can bespecified, the resolution can be increased and the efficiency ofpattern-matching processing can be improved by generatingpattern-matching images from interpolated images and performing patternmatching again only in the vicinity of the position.

FIG. 3A and FIG. 3B show the states in which left and rightpattern-matching images have been generated, respectively, by performingpixel interpolation on small areas of 3×3 pixel region containing thepixel surrounding the position, after the position is specifiedapproximately. FIG. 3A shows the left pattern-matching image and FIG. 3Bshows the right pattern-matching image, respectively. FIG. 3 shows acase where matching is performed in the transverse direction.

The pixel interpolation technique for generating the left and rightpattern-matching images shown in FIG. 3A and FIG. 3B, respectively, isthe same as that in the case shown in FIG. 1. The pattern-matchingimages shown in FIG. 3A and FIG. 3B are generated based on the front andrear pixels in the transverse direction in accordance with pixel q₁₂corresponding to the specified position. In the left pattern-matchingimage shown in FIG. 3A, the pixel corresponding to the specifiedposition is p₂₂.

In matching between the left and right patterns, in the case where theleft pattern-matching image shown in FIG. 3A is taken as the referenceimage, first nine pixels b₁₁, p₂₁, b₂₁, b₁₂, a₂₁, b₂₂, b₁₃, p₂₂ and b₂₃,which form the 3×3 pixel pattern of the left pattern matching image, arecompared with nine pixels d₁₁, q₂₁, d₂₁, d₁₂, c₂₁, d₂₂, d₁₃, q₂₂ andd₂₃, which form a comparison image, that is, the 3×3 pixel pattern ofthe right pattern-matching image, and correlation coefficients arefound.

Next, nine pixels b₁₁, p₂₁, b₂₁, b₁₂, a₂₁, b₂₂, b₁₃, p₂₂ and b₂₃, whichform the 3×3 pixel pattern of the left pattern-matching image arecompared with nine pixels d₁₂, c₂₁, d₂₂, d₁₃, q₂₂, d₂₃, d₁₄, c₂₂ andd₂₄, which form the right pattern-matching image, that is, the nextcomparison image, and correlation coefficients are found.

The second next comparison image is generated from the rightpattern-matching image and correlation coefficients are found. Then, thenext reference image is generated from the left pattern-matching imageand matching is performed with the comparison images generatedsequentially from the right pattern-matching image. In this manner, thematching position is specified by the least value among the obtainedcorrelation coefficients.

As pattern matching is performed using the pattern-matching imagesinterpolated with pixels on only the vicinity to the specified matchingposition, which has been obtained in a normal pattern matching using the3×3 pixel patterns generated from the original image, the resolution canbe improved without a large amount of calculation and time beingrequired.

In generating the left and right pattern-matching images shown in FIG.3A and FIG. 3B according to the third embodiment, pixel interpolation isperformed on both the left pattern-matching image, which is thereference image, and the right pattern-matching image, which is acomparison image but, in the fourth embodiment shown in FIG. 4, thenine-pixel pattern cut out from the left original image, which is usedin a normal pattern matching, is used as the left pattern-matchingimage, that is, the reference image without any changes made to it, asshown in FIG. 4A, and pixel interpolation is performed only on the rightpattern-matching image, from which a comparison image is generated, asshown in FIG. 4B.

In the left and right pattern matching, nine pixels p₁₁, p₂₁, p₃₁, p₁₂,p₂₂, p₃₂, p₁₃, p₂₃ and p₃₃, forming the 3×3 pixel pattern of the leftpattern-matching image, are first compared with nine pixels d₁₁, q₂₁,d₂₁, d₁₂, c₂₁, d₂₂, d₁₃, q₂₂ and d₂₃, forming the 3×3 pixel pattern ofthe right pattern-matching image, which is used as a comparison image,and correlation coefficients are found.

Then, while the left pattern-matching image, which is used as thereference image, is left as is, it is compared with nine pixels d₁₂,c₂₁, d₂₂, d₁₃, q₂₂, d₂₃, d₁₄, c₂₂ and d₂₄, which form the nextcomparison image shifted by one pixel in the right pattern-matchingimage, and correlation coefficients are found.

From the right pattern-matching image, the second next comparison image,the third one, and so on, are generated sequentially, and correlationcoefficients are found. The position corresponding to the least valueamong thus obtained correlation coefficients is judged to be thematching position.

As pattern matching is performed using the pattern-matching imagesinterpolated with pixels only for generating the comparison images inthe vicinity of the specified matching position, which has been obtainedby a normal pattern matching of the 3×3 pixel patterns generated fromthe original image, the amount of calculations and the period of timecan be further reduced and the resolution can also be further improvedcompared to the pattern-matching processing shown in FIG. 3.

In the pattern-matching processing in the first to fourth embodimentsdescribed above, what is focused on is how to perform pixelinterpolation on the pixel patterns cut out from the left and rightoriginal images when the left and right pattern-matching images aregenerated, and the applied pixel interpolation is basically theconventional pixel interpolation technique shown in FIG. 10. In theconventional pixel interpolation technique, the interpolation pixelvalue is a pixel value calculated by simply averaging the values of theperipheral pixels around the position to be interpolated with pixels.However, there is a possibility that the interpolation pixel valueobtained by simply averaging the peripheral pixels according to theconventional pixel interpolation technique will not reflect the actualcase.

Below is described the embodiments relating to the pixel interpolationtechnique in which the influence of errors due to pixel interpolation islessened by improving the method for averaging peripheral pixels in theconventional pixel interpolation method.

FIG. 5 shows the case where a left pattern-matching image of 5×5 pixelsis generated by applying a first pixel interpolation technique accordingto the present embodiment to the 3×3 pixel pattern cut out from the leftoriginal image shown in FIG. 10A.

The manner, in which pixel interpolation is performed between twotransversely adjacent pixels in the 3×3 pixel pattern, is the same asthe conventional pixel interpolation technique, that is, from theaverage value of pixel p₁₁ and pixel p₁₂, interpolation pixel a₁₁ isobtained, and in the same manner, interpolation pixels a₁₂ to a₃₂ areobtained sequentially in the transverse direction.

Then, pixel interpolation is performed between two vertically adjacentpixels in the 3×3 pixel pattern. In the conventional pixel interpolationtechnique, the interpolation start position to be interpolated withpixels is pixel position e₁₁ at the end of, for example, the pixelpattern shown in FIG. 5, but in the pixel interpolation techniqueaccording to the present embodiment, the start position at which pixelinterpolation is performed between two vertically adjacent pixels ispixel position e₁₃. This is because, if interpolation is started frompixel position e₁₁, the number of pixels for averaging is small and,therefore, interpolation is stared from a position at which the numberof pixels for averaging is large.

First, the pixel value to be interpolated to the pixel position e₁₃ isobtained from the following expressione ₁₃=(a ₁₁ +p ₁₂ +a ₁₂ +a ₂₁ +p ₂₂ +a ₂₂)/6Next, the pixel value to be interpolated to pixel position e₁₂ isobtained from the following expressione ₁₂=(p ₁₁ +a ₁₁ +p ₁₂ +e ₁₃ +p ₂₁ +a ₂₁ +p ₂₂)/7Next, the pixel value to be interpolated to pixel position e₁₁ isobtained from the following expressione ₁₁=(p ₁₁ +a ₁₁ +e ₁₂ +a ₂₁ +p ₂₁)/5Then, the pixel value for pixel position e₁₄ is obtained. In thismanner, interpolation pixel values are calculated sequentially startingfrom the position at which the number of pixels for averaging is large.The same processing is applied to other pixel positions e₂₁ to e₂₅ inthe transverse direction.

In the pixel interpolation technique according to the present embodimentdescribed above, the interpolation start position at which pixelinterpolation is performed between two vertically adjacent pixels is apixel position at which the number of pixels for averaging is large. Theaverage value is a simple average of plural pixel values, but in asecond pixel interpolation technique, which is the next embodiment, theinterpolation pixel values contained in the plural target pixel valuesfor averaging are weighted in order to lessen the influence of errorsdue to interpolation pixels as significantly as possible. This isdesigned to suppress the influence of errors inherent in interpolationpixels because, if the pixels including interpolation pixels are usedfor averaging without any changes made to them, it means that thesepixels are regarded as the actually existent pixels, as a result. Aconcrete example of a case, where the interpolation pixel values areweighted when calculated, is shown in FIG. 6.

FIG. 6 shows the case where the 3×3 pixel pattern of the left originalimage shown in FIG. 10A is interpolated with pixels, and the pixelinterpolation between two transversely adjacent pixels is the same asthat used to generate the left pattern-matching image shown in FIG. 5.The interpolation procedure for performing pixel interpolation betweentwo vertically adjacent pixels is the same as that used to generate thepattern-matching image shown in FIG. 5. For example, the interpolationpixel value of pixel interpolation position f₁₃ is obtained from thefollowing expressionf ₁₃=(a ₁₁/2+p ₁₂ +a ₁₂2+a ₁₂/2+p ₂₂ +a ₂₂/2)/4where interpolation pixel values a₁₁, a₁₂, a₂₁ and a₂₂ are assigned witha weight of ½. Due to the assignment of weight of ½, the number ofpixels for averaging is decreased from six to four.

The assignment of weight of ½ adopted here is only an example. All thatis required is the assignment of weight should decrease the values ofpixels that really exist, and therefore the weight should be less thanone.

In the second pixel interpolation technique, as described above, thecontained interpolation pixel values are obtained by assigning a weightto the pixels that really exist in averaging for performing pixelinterpolation between two vertically adjacent pixels and, therefore, thereliability of the pattern-matching processing of the generatedpattern-matching images can be improved.

In the first and second pixel interpolation techniques described above,when pixel interpolation is performed between two vertically adjacentpixels, the average value of the eight peripheral pixels is used as theinterpolation pixel value. Contrary to this, a third pixel interpolationtechnique, which will be described below by reference to FIG. 7, adoptsa further simplified procedure for performing pixel interpolationbetween two adjacent pixels.

FIG. 7 shows the case where the 3×3 pixel pattern of the left originalimage shown in FIG. 10A is interpolated with pixels. The method forobtaining the interpolation pixels a₁₁, a₁₂, a₂₁, a₂₂, a₃₁ and a₃₂between two transversely adjacent pixels is the same as that used togenerate the left pattern-matching image shown in FIG. 5, but when pixelinterpolation is performed between two vertically adjacent pixels in theleft original image, for example, when pixel interpolation is performedbetween pixel p₁₁ and pixel p₂₁, interpolation pixel value g₁₁ isobtained from the following expressiong ₁₁=(p ₁₁ +p ₂₁)/2In the same manner, the interpolation pixel values of interpolationpositions g₁₃, g₁₅, g₂₁, g₂₃ and g₂₅ are obtained based on the pixelsthat really exist on the upper and lower sides thereof.

Next, when the interpolation pixel values of interpolation pixelpositions h₁₂, h₁₄, h₂₂ and h₂₄ to be interpolated with pixels areobtained, the average value of the four pixels, which are located on theupper and lower sides of and to the left-hand and right-hand sides ofeach interpolation pixel position, is calculated. For example, when theinterpolation pixel value of pixel position h₁₂ is obtained bycalculation using the following expressionh ₁₂=(a ₁₁ +f ₁₁ +f ₁₃ +a ₂₁)/4The interpolation pixel values of other pixel positions can becalculated in the same manner.

According to the third pixel interpolation technique, as describedabove, the period of time required for pattern-matching processing canbe reduced compared to the first and second pixel interpolation means,because a technique of averaging four, that is, the upper, lower, leftand right pixels, is adopted instead of a technique of averaging eightperipheral pixels for pixel interpolation between two verticallyadjacent pixels in the original image.

Moreover, it is possible to introduce the idea of weighted averagingused in the second pixel interpolation technique shown in FIG. 6 intothe third pixel interpolation technique for generating thepattern-matching image shown in FIG. 7 to establish a fourth pixelinterpolation technique. In the fourth pixel interpolation technique, itis designed so that errors are avoided in averaging when theinterpolation pixel values of interpolation pixel positions h₁₂, h₁₄,h₂₂ and h₂₄ are obtained by assigning specified weights, because theupper, lower, left and right pixels are interpolation pixels that do notreally exist.

For example, the weight is assumed to be ½, the interpolation pixelvalue of pixel position h₁₂ can be obtained by the following expressionh ₁₂=(a ₁₁/2+f ₁₁/2+f ₁₃/2+a ₂₁/2)/2The interpolation pixel values relating to other pixel positions can beobtained in the same procedure.

In the first to fourth pixel interpolation techniques described above,the left and right pattern-matching images are generated by performingpixel interpolation between two pixels among nine pixels making up the3×3 pixel patterns shown in FIG. 10A and FIG. 10B, which have been cutout from the left and right original images photographed by a stereocamera.

However, when the original image photographed by a stereo camera isphotographed by the interlaced scanning method, pixels are arranged inthe transverse direction at every two vertical pixel positions in theoriginal image in a frame. In other words, the next pixel row isphotographed in the next frame, therefore, the period of time requiredfor two frames is necessary to obtain the transverse pixel arrayscorresponding to a display image of the original image. A fifth pixelinterpolation technique is one in which pixel interpolation is performedusing only the transverse components, without using the vertical pixelcomponents, when detection accuracy is not required in the verticaldirection in the original image. An example of generation of the leftpattern-matching image according to the fifth pixel interpolationtechnique is shown in FIG. 8.

In the fifth pixel interpolation technique, pixel interpolation isperformed only between two transversely adjacent pixels in one of theoriginal image photographed by the interlaced scanning method,therefore, six interpolation pixels a₁₁, a₁₂, a₂₁, a₂₂, a₃₁ and a₃₂ haveto be obtained based on pixels p₁₁ to p₃₃ of the 3×3 pixel pattern.

For example, interpolation pixel value a₁₁ can be obtained using pixelp₁₁ and pixel p₁₂ from the following expressiona ₁₁=(p ₁₁ +p ₁₂)/2Other interpolation pixel values can be obtained from the sameexpression using two adjacent pixel values.

According to the fifth pixel interpolation technique, as describedabove, when the original image is one by the interlaced scanning method,pattern-matching images can be generated by a simple method, therefore,the processing time can be considerably reduced while the detectionaccuracy in the transverse direction is maintained.

By applying the first to fifth pixel interpolation techniques, describedabove, to a case where left and right pattern-matching images aregenerated from left and right original images photographed by a stereocamera, processing time can be reduced while the resolution in thepattern-matching processing is maintained.

According to the present invention, as described above, when patternmatching is performed, a pattern-matching processing method is adoptedin which the original image is interpolated with pixels andpattern-matching images are generated, therefore, the period of timerequired for pattern-matching processing can be reduced while theresolution of the images is maintained, and as a result, it is no longernecessary to provide a storage memory to store the pattern-matchingimages, which are the original images interpolated with pixels.

Moreover, by applying the pattern-matching processing method accordingto the present invention to a stereoscopic distance measuring system,the accuracy in distance measurement can be improved.

In the pattern-matching processing method according to the presentinvention, after the parallax position is specified approximately,pattern-matching images regarding the position are generated by pixelinterpolation and then pattern matching is performed, therefore, forexample, only 100 images of 3×3×2 have to be processed compared to acase where two 640×480 images must be processed in a conventional pixelinterpolation technique, and as a result, the processing time requiredfor replacing the whole of the original image with the imagesinterpolated with pixels can be reduced.

In addition, in the pattern-matching processing method according to thepresent invention, after the parallax position is specifiedapproximately, only a right pattern-matching image regarding theposition is generated by pixel interpolation and then pattern matchingis performed, therefore, for example, only 100 images of 3×3 have to beprocessed compared to a case where a 640×480 image has to be processedin a conventional pixel interpolation technique, and as a result, theprocessing time required for replacing the whole of the original imagewith the images interpolated with pixels can be reduced.

Moreover, in the pattern-matching processing method according to thepresent invention, when pixel interpolation is performed on the pixelpatterns cut out from the original images and pattern-matching imagesare generated, interpolation pixel values are obtained by weighting theperipheral pixels and averaging them, therefore, errors due to pixelinterpolation can be lessened.

In the pattern-matching processing method according to the presentinvention, when pixel interpolation is performed on the pixel patternscut out from the original images and pattern-matching images aregenerated, interpolation pixel values are obtained as an average valuecalculated from only the upper, lower, left and right pixels that reallyexist, therefore, the range in which pixel interpolation can beperformed is widened and an accurate pixel interpolation can beperformed.

Particularly, when the image photographed by a stereo camera is one bythe interlaced scanning method, pixel interpolation is performed basedon the pixels that really exist on the left and right sides andpattern-matching images are generated, therefore, an accurate pixelinterpolation can be realized.

By using an image processing apparatus, in which the above-mentionedpattern-matching processing method according to the present inventionhas been applied to a stereoscopic distance measuring system, thedistance to an object ahead of the vehicle can be detected with accuracyand the pattern-matching processing time required for detection can bereduced, therefore, it is possible to swiftly provide accurate distanceinformation for driving support and warning and safe driving of thevehicle is promoted.

In this specification, description is made with the left-hand side andthe right-hand side being clearly defined such as a left image, rightimage, left area, right area, and so on, to give a clear description andavoid complexity, but the operations are the same even if the left-handside and the right-hand side are switched, and it is clear that such acase is also included in the present invention.

1. A pattern-matching processing method, comprising: an area generatingstep for generating a left area and a right area each having a givenrange from a left image and a right image photographed in stereo; apixel interpolating step for generating an interpolation pixel betweenpixels of a plurality of pixels included respectively in the left areaand the right area, wherein the pixel interpolating step includesperforming a first pixel interpolation between two first adjacent pixelsalong a first direction in the left area and in the right area, andperforming a second pixel interpolation between two second adjacentpixels along a second direction perpendicular to the first direction,wherein the second pixel interpolation includes calculating an averagevalue of plural pixels surrounding a position to be interpolated,wherein the plural pixels include both interpolated pixels andnon-interpolated pixels, and wherein in calculating the average value,the interpolated pixels among the plural pixels surrounding the positionto be interpolated are assigned a weight less than the weight given tothe non-interpolated pixels; and a pattern matching step for performingpattern matching using the left area interpolated with the correspondinginterpolation pixels and the right area interpolated with thecorresponding interpolation pixels.
 2. A pattern-matching processingmethod, as set forth in claim 1, wherein said first and second pixelinterpolations are performed on said left area and said right areacontaining a number of pixels surrounding a matching position specifiedby the pattern matching step based on an original left area and anoriginal right area.
 3. A pattern-matching processing method,comprising: an area generating step for generating a left area and aright area each having a given range from a left image and a right imagephotographed in stereo; a pixel interpolating step for generating aninterpolation pixel between pixels of a plurality of pixels included ineither the left area or the right area, wherein the pixel interpolatingstep includes performing a first pixel interpolation between two firstadjacent pixels along a first direction in the left area or in the rightarea, and performing a second pixel interpolation between two adjacentpixels along a second direction perpendicular to the first direction,wherein the second pixel interpolation includes calculating an averagevalue of plural pixels surrounding a position to be interpolated,wherein the plural pixels include both interpolated pixels andnon-interpolated pixels, and wherein in calculating the average value,the interpolated pixels among the plural pixels surrounding the positionto be interpolated are assigned a weight less than the weight given tothe non-interpolated pixels; and a pattern matching step for performingpattern matching using the area interpolated with the correspondinginterpolation pixel and the area without interpolation pixels.
 4. Apattern-matching processing method, as set forth in claim 3, whereinsaid first and second interpolations are performed on said left area andsaid right area containing a number of pixels surrounding a matchingposition specified by the pattern matching step based on an originalleft area and an original right area.
 5. A pattern-matching processingmethod, as set forth in claim 1, wherein said second pixel interpolationis performed starting from a pixel position at which a number of thepixels surrounding the position to be interpolated is largest, for whichthe average value is calculated.
 6. An image processing apparatusmeasuring a distance to an object that is photographed as images, byperforming pattern-matching processing based on left and right imagesphotographed by a stereo camera, comprising: an area generating unit forgenerating a left area and a right area each having a fixed range fromthe left image; a pixel interpolating unit for generating aninterpolation pixel between pixels of a plurality of pixels includedrespectively in the left area and the right area, wherein the pixelinterpolating step includes performing a first pixel interpolationbetween two first adjacent pixels along a first direction in the leftarea and in the right area, and performing a second pixel interpolationbetween two second adjacent pixels along a second directionperpendicular to the first direction, wherein the second pixelinterpolation includes calculating an average value of plural pixelssurrounding a position to be interpolated, wherein the plural pixelsinclude both interpolated pixels and non-interpolated pixels, andwherein in calculating the average values, the interpolated pixels amongthe plural pixels surrounding the position to be interpolated areassigned a weight less than the weight given to the non-interpolatedpixels; and a pattern-matching processing unit having a pattern-matchingunit performing pattern matching based on the left area interpolatedwith the corresponding interpolation pixels and the right areainterpolated with the corresponding interpolation pixels, or based onone of the areas without pixel interpolation and the other areainterpolated with the corresponding interpolation pixels.
 7. An imageprocessing apparatus, as set forth in claim 6, wherein said pixelinterpolating unit performs pixel interpolation on the left area and theright area containing a number of pixels surrounding a matching positionspecified by the pattern matching step based on an original left areaand an original right area.
 8. An image processing apparatus, as setforth in claim 6, wherein said pixel interpolating unit performs pixelinterpolation on the right area containing a number of pixelssurrounding a matching position specified by the pattern matching stepbased on an original left area and an original right area.
 9. An imageprocessing apparatus, as set forth in claim 6, wherein said pixelinterpolating unit performs pixel interpolation starting from a pixelposition at which a number of the pixels surrounding the position to beinterpolated is largest, for which the average value is calculated. 10.A pattern-matching processing method, as set of forth in claim 3 or 4,wherein said second pixel interpolation is performed starting from apixel position at which a number of the pixels surrounding a position tobe interpolated is largest, for which the average value is calculated.11. An image processing apparatus measuring a distance to an object thatis photographed as images, by performing pattern-matching processingbased on left and right images photographed by a stereo camera,comprising; an area generating unit for generating a left area and aright area each having a fixed range from the left image; a pixelinterpolating unit for generating an interpolation pixel between pixelsof a plurality of pixels included in respectively the left area and theright area, wherein the pixel interpolating step includes performingfirst pixel interpolation between two first adjacent pixels along afirst direction in the left area and in the right area, and performing asecond pixel interpolation between two second adjacent pixels along asecond direction perpendicular to the first direction, wherein thesecond pixel interpolation includes calculating an average value ofplural pixels surrounding a position to be interpolated, wherein theplural pixels include both interpolated pixels and non-interpolatedpixels, and wherein in calculating the average value, the alreadyinterpolated pixels among the plural pixels surrounding the position tobe interpolated are assigned a weight less than the weight given to thenon-interpolated pixels; and a pattern-matching processing unit having apattern-matching unit performing pattern matching based on the left areainterpolated with the corresponding interpolation pixels and the rightarea interpolated with the corresponding interpolation pixels, or basedon one of the areas without pixel interpolation and the other areainterpolated with the corresponding interpolation pixels; and a distancemeasuring unit for calculating the distance from the difference inpositions of the left image and the right image based on a matchingposition specified by performing pattern matching on the left area andthe right area.